We review and explore stationarity of different models.

1 MA(q)

• This is always stationary.
• ACF ${\displaystyle \rho (h)=0}$, when ${\displaystyle |h|>q}$.

2 AR(p)

Now we discuss its stationarity.

2.1 p=1

Now ${\displaystyle y_t-{\varphi }_1{y}_{t-1}={\varphi }_0+{\epsilon }_t}$.

1. ${\displaystyle |{\varphi }_1|<1}$: $$\begin{array}{}\text{(2.2)}& y_t=\frac{{\varphi }_0}{1-{\varphi }_1}+\sum _{j=0}^{\mathrm{\infty }}{\varphi }_1^j{\epsilon }_{t-j}.\end{array}$$ This is well-defined because ${\displaystyle |{\varphi }_1|<1}$. ${\displaystyle {\epsilon }_t}$ is independent of ${\displaystyle y_{t-1},y_{t-2},\cdots }$. We refer this to causal stationary ${\displaystyle \mathrm{AR}(1)}$.
2. ${\displaystyle |{\varphi }_1|>1}$: $$\begin{array}{}\text{(2.3)}& y_t=\frac{{\varphi }_0}{1-{\varphi }_1}-\sum _{j=0}^{\mathrm{\infty }}\frac{{\epsilon }_{t+j}}{{\varphi }_1^j}.\end{array}$$ Now ${\displaystyle {\epsilon }_t}$ is not independent of ${\displaystyle y_{t-1},y_{t-2},\cdots }$, but independent of ${\displaystyle y_{t+1},y_{t+2},\cdots }$. Refer this to non-causal stationary ${\displaystyle \mathrm{AR}(1)}$.
3. ${\displaystyle |{\varphi }_1|=1}$: if ${\displaystyle {\varphi }_1=1}$, ${\displaystyle y_t-y_{t-1}={\varphi }_0+{\epsilon }_t}$, this means ${\displaystyle y_t-y_{t-1}}$ is Gaussian white noise. When ${\displaystyle {\varphi }_0=0}$, this is Random Walk model. There is no stationary solution.

Now we use backshift notation to derive (2.2) and (2.3): ${\displaystyle B^k{y}_t=y_{t-k},B^0=1}$. So ${\displaystyle \mathrm{AR}(1)}$ becomes ${\displaystyle \varphi (B)y_t={\varphi }_0+{\epsilon }_t}$, where ${\displaystyle \varphi (z)=1-{\varphi }_{1}z,\varphi (B)=1-{\varphi }_{1}B}$. So $$y_t=\frac{1}{\varphi (B)}({\varphi }_0+{\epsilon }_t),$$ and note that $$\frac{1}{\varphi (B)}=\frac{1}{1-{\varphi }_{1}B}=\sum _{j=0}^{\mathrm{\infty }}{\varphi }_1^j{B}^j,$$ so $$y_t=\sum _{j=0}^{\mathrm{\infty }}{\varphi }_1^j{B}^j({\varphi }_0)+\sum _{j=0}^{\mathrm{\infty }}{\varphi }_1^j{B}^j({\epsilon }_t)=\sum _{j=0}^{\mathrm{\infty }}{\varphi }_1^j+\sum _{j=0}^{\mathrm{\infty }}{\varphi }_1^j{\epsilon }_{t-j}.$$ This only makes sense when ${\displaystyle |{\varphi }_1|<1}$, so $$y_t=\frac{{\varphi }_0}{1-{\varphi }_1}+\sum _{j=0}^{\mathrm{\infty }}{\varphi }_1^j{\epsilon }_{t-j},$$ this gives us (2.2).

And when ${\displaystyle |{\varphi }_1|>1}$, note that $$\frac{1}{1-r}=-\frac{1}{r}\frac{1}{1-(1/r)}=-\frac{1}{r}(1+\frac{1}{r}+\frac{1}{r^2}+\cdots )=-\sum _{j=1}^{\mathrm{\infty }}{r}^{-j}.$$
So ${\displaystyle \frac{1}{\varphi (B)}=-\sum _{j=1}^{\mathrm{\infty }}\frac{B^{-j}}{{\varphi }_1^j}}$, and $$\frac{1}{\varphi (B)}({\varphi }_0+{\epsilon }_t)=-{\varphi }_0\sum _{j=1}^{\mathrm{\infty }}\frac{1}{{\varphi }_1^j}-\sum _{j=1}^{\mathrm{\infty }}\frac{B^{-j}{\epsilon }_t}{{\varphi }_1^j}=\frac{{\varphi }_0}{1-{\varphi }_1}-\sum _{j=1}^{\mathrm{\infty }}\frac{{\epsilon }_{t+j}}{{\varphi }_1^j}$$ which is (2.3).

To recap, $$\begin{array}{}\text{(2.4)}& \frac{1}{1-{\varphi }_{1}B}=\sum _{j=0}^{\mathrm{\infty }}{\varphi }_1^j{B}^j,\frac{1}{1-{\varphi }_{1}B}=-\sum _{j=1}^{\mathrm{\infty }}\frac{B^{-j}}{{\varphi }_1^j}.\end{array}$$

• ${\displaystyle |{\varphi }_1|<1}$, use first formula.
• ${\displaystyle |{\varphi }_1|>1}$, use second formula.
• ${\displaystyle |{\varphi }_1|=1}$, (2.4) don't make sense.

2.2 p≥1

Still use backshift notation: $$\varphi (B)y_t={\varphi }_0+{\epsilon }_t,\varphi (B)=1-{\varphi }_{1}B-\cdots -{\varphi }_p{B}^p.$$ To use (2.4), we need to factorize the polynomial $$\varphi (z)=1-{\varphi }_{1}z-\cdots -{\varphi }_p{z}^p=(1-a_{1}z)\cdots (1-a_{p}z),$$ so ${\displaystyle \varphi (B)=(1-a_{1}B)\cdots (1-a_{p}B)}$.
We then get $$\begin{array}{rl}{y}_t& =\frac{1}{(1-a_{1}B)\cdots (1-a_{p}B)}({\varphi }_0+{\epsilon }_t)=\prod _{k=1}^p\frac{1}{1-a_{k}B}({\varphi }_0+{\epsilon }_t)\\ & =\prod _{k:|a_k|<1}\left(\sum _{j=0}^{\mathrm{\infty }}{a}_k^j{B}^j\right)\prod _{k:|a_k|>1}\left(\sum _{j=1}^{\mathrm{\infty }}\frac{B^{-j}}{a_k^j}\right)({\varphi }_0+{\epsilon }_t).\end{array}$$

1. If every ${\displaystyle k}$, ${\displaystyle |a_k|<1}$, then $$\begin{array}{rl}{y}_t& =\prod _k\left(\sum _{j=0}^{\mathrm{\infty }}{a}_k^j{B}^j\right)({\varphi }_0+{\epsilon }_t)\\ & =(\sum _{j_1=0}^{\mathrm{\infty }}\cdots \sum _{j_p=0}^{\mathrm{\infty }}{a}_1^{j_1}\cdots a_p^{j_p}{B}^{j_1+\cdots +j_p})({\varphi }_0+{\epsilon }_t)\\ & ={\varphi }_0\sum _{j_1=0}^{\mathrm{\infty }}\cdots \sum _{j_p=0}^{\mathrm{\infty }}{a}_1^{j_1}\cdots a_p^{j_p}+\sum _{j_1=0}^{\mathrm{\infty }}\cdots \sum _{j_p=0}^{\mathrm{\infty }}{a}_1^{j_1}\cdots a_p^{j_p}{\epsilon }_{t-j_1-\cdots -j_p}.\end{array}$$
   This is a causal stationary solution. By collecting ${\displaystyle j_1+\cdots +j_p=j}$ for ${\displaystyle j=0,1,\cdots }$, $$y_t=\mu +\sum _{j=0}^{\mathrm{\infty }}{\psi }_j{\epsilon }_{t-j}.$$
2. If every ${\displaystyle k}$, ${\displaystyle |a_k|>1}$, similarly $$y_t=\mu +\sum _{j=-\mathrm{\infty }}^{\mathrm{\infty }}{\psi }_j{\epsilon }_{t-j}.$$
3. If every ${\displaystyle k}$, ${\displaystyle |a_k|=1}$, there is no stationary solution.

2.3 The Box-Jenkins Modeling Philosophy

• Only work with causal stationary AR models (ARMA later)
• If the data are not fitting this, difference it to fit causal stationary ARMA models.

3 ARMA(p, q) Models

Combine AR and MA models: $$\begin{array}{}\text{(3.1)}& (y_t-\mu )-{\varphi }_1(y_{t-1}-\mu )-\cdots -{\varphi }_p(y_{t-p}-\mu )={\epsilon }_t+{\theta }_1{\epsilon }_{t-1}+\cdots +{\theta }_q{\epsilon }_{t-q}.\end{array}$$

When ${\displaystyle p=0}$, this is ${\displaystyle \mathrm{MA}(q)}$; when ${\displaystyle q=0}$, this is ${\displaystyle \mathrm{AR}(p)}$.

As usual ${\displaystyle {\epsilon }_t\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2)}$. The parameters here are ${\displaystyle \mu ,{\varphi }_1,\cdots ,{\varphi }_p,{\theta }_1,\cdots ,{\theta }_q,\sigma }$. In backshift notation: $$\varphi (B)(y_t-\mu )=\theta (B){\epsilon }_t,$$ where ${\displaystyle \varphi (z)=1-{\varphi }_{1}z-\cdots -{\varphi }_p{z}^p,\theta (z)=1+{\theta }_{1}z+\cdots +{\theta }_q{z}^q}$.

It can be shown that: if ${\displaystyle \varphi (z)}$ has all roots with modulus strictly larger than ${\displaystyle 1}$, then ${\displaystyle \mathrm{ARMA}(p,q)}$ has a stationary causal solution: $$y_t=\mu +{\psi }_0{\epsilon }_t+{\psi }_1{\epsilon }_{t-1}+\cdots$$
Denote ${\displaystyle \psi (z)={\psi }_0+{\psi }_{1}z+{\psi }_2{z}^2+\cdots }$, we can get coefficients by ${\displaystyle \psi (z)=\theta (z)/\varphi (z)}$, i.e. $$\begin{array}{rl}\theta (z)& =1+{\theta }_{1}z+\cdots +{\theta }_q{z}^q=\varphi (z)\psi (z)\\ & =(1-{\varphi }_{1}z-\cdots -{\varphi }_p{z}^p)({\psi }_0+{\psi }_{1}z+{\psi }_2{z}^2+\cdots ),\end{array}$$ and then comparing coefficients of ${\displaystyle z^j}$ on both sides: $$1={\psi }_0,{\theta }_1={\psi }_1-{\psi }_0{\varphi }_1,{\theta }_2={\psi }_2-{\psi }_1{\varphi }_1-{\psi }_0{\varphi }_2,\cdots$$
${\displaystyle \mathrm{ARMA}(p,q)}$ generalizes both ${\displaystyle \mathrm{AR}(p)}$ and ${\displaystyle \mathrm{MA}(q)}$ models. When ${\displaystyle p=q=0}$, we obtain the white noise model. When ${\displaystyle p=0}$, we get ${\displaystyle \mathrm{MA}(q)}$; when ${\displaystyle q=0}$, we get ${\displaystyle \mathrm{AR}(p)}$.
For ARMA, ACF and PACF are more complicated. Neither ACF nor PACF cuts off after a certain lag, if both ${\displaystyle p,q\ge 1}$.